Repository navigation
fold U+0345 after canonical ordering in utf8proc_decompose_custom - #379
Open
shamsManna wants to merge 1 commit into
Open
shamsManna wants to merge 1 commit into
shamsManna wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
utf8proc_decompose_custom case-folds each codepoint before the canonical ordering, but U+0345 (ccc 240) folds to U+03B9, a starter, so marks after it end up on the iota instead of the base letter and canonically equivalent strings fold differently: utf8proc_NFKC_Casefold maps U+1FB3 U+0301 to U+03B1 U+03AF but its NFC form U+1FB4 to U+03AC U+03B9 (fixes #3). this leaves U+0345 unfolded, also inside the 63 precomposed letters that decompose to it, and folds it once its run of marks ends at a starter of the canonical decomposition (dropped ignorables count), so the output matches the UCD's NFC(NFKC_CF(NFD(x))) on every codepoint and on 750k random strings and is byte-identical to before for input without U+0345. CHARBOUND is left as it was since its sentinels block the reordering, and there's a regression test in test/misc.c.